Papers with concrete task
Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks (2026.acl-short)
Copied to clipboard
| Challenge: | Xiao et al., 2024) show that softmax models display an attention sink . he argues that normalization over a probability simplex must force attention to collapse onto a stable anchor to realize a default state. |
| Approach: | They show that normalization over a trigger-conditional behavior *necessarily* induces a sink in softmax self-attention models. |
| Outcome: | The proposed model can solve a task with no sink in softmax models. |